Article reading device

ABSTRACT

An article reading device according to an embodiment includes a display device and an image capturing device that generates an image of an article. A processor extracts, from the image, first feature data for recognizing the article and second feature data for determining whether to recognize the article based on the first feature data. The processor determines whether to recognize the article. If it is determined to recognize the article, the processor recognizes the article based on the extracted first feature data, and controls the display device to display a recognition result. If it is determined to not recognize the article, extract a barcode from the image, the processor identifies the article based on the extracted barcode, and control the display device to display an identification result. The processor performs a transaction settlement with respect to the recognition result, if any, and the identification result, if any.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/818,341, filed Mar. 13, 2020, which is a continuation of U.S. patentapplication Ser. No. 15/691,319, filed Aug. 30, 2017, now abandoned,which is based upon and claims the benefit of priority from JapanesePatent Application No. 2016-171795, filed Sep. 2, 2016, the entirecontents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to an article readingdevice.

BACKGROUND

In the related art, when merchandise is to be sold at a store, asalesperson registers the merchandise by reading a barcode attached tothe merchandise. In addition, if the barcode is not attached to themerchandise, the salesperson compares the merchandise with a pluralityof selection buttons corresponding to respective merchandise itemsdisplayed on a screen, operates a selection button corresponding to themerchandise, and registers the merchandise. On the other hand, there isalso a technique in which merchandise registration is performed usinggeneric object recognition for recognizing types or the like ofmerchandise by extracting feature data of the merchandise from capturedimages obtained by imaging the merchandise and by comparing theextracted feature data with verification data prepared in advance.Accordingly, a device which can execute both of merchandise registrationbased on a barcode and merchandise registration based on an objectrecognition result without performing an operation such as switching hasbeen developed.

As described above, in an article reading device which can execute thebarcode reading and object recognition without performing the switchingoperation, it is desirable to simultaneously display a selection buttoninstructing merchandise registration and an image of merchandiseactually captured on a screen, in order to increase efficiency ofmerchandise registration. However, since two different pieces ofinformation are displayed in one screen, respective display ranges arereduced and it is difficult to view display contents. Accordingly, thereis a problem that efficiency of a registration operation is reduced.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is an exterior perspective view illustrating an example of acheck-out system including an article reading device according to anembodiment.

FIG. 2 is a block diagram illustrating an example of a hardwareconfiguration of the check-out system.

FIG. 3 illustrates an example configuration of a PLU file.

FIG. 4 is a block diagram illustrating an example of a functionalconfiguration of the check-out system.

FIG. 5A illustrates an example of a pre-set screen.

FIG. 5B illustrates an example of a selection screen on which aselection button of a merchandise candidate is displayed together with acaptured image of an article.

FIG. 5C illustrates an example of a reading result screen displayed whencontents of a barcode attached to merchandise are read by a barcodereading process.

FIG. 5D illustrates an example of a re-recognition selection screendisplayed in a case where the merchandise candidate is not uniquelydetermined.

FIG. 6 is a state transition diagram illustrating an example of screentransitions.

FIG. 7 is a sequence diagram illustrating a process example forperforming one transaction.

FIG. 8 is a flow chart illustrating an example sequence of operationsfor a series of merchandise registration processes.

FIG. 9 is a flow chart illustrating a modified example sequence ofoperations for the series of merchandise registration processes.

DETAILED DESCRIPTION

Embodiments provide an article reading device capable of simultaneouslyexecuting barcode reading and object recognition so that a capturedimage of merchandise can be displayed for easy viewing withoutperforming a special operation as necessary.

An article reading device according to an embodiment includes a displaydevice and an image capturing device that generates an image of anarticle. A processor receives the generated image of the article andextracts from the image, first feature data for recognizing the articleand second feature data for determining whether to recognize the articlebased on the first feature data. The processor determines, based on theextracted second feature data, whether to recognize the article. If itis determined to recognize the article, the processor recognizes thearticle based on the extracted first feature data, and controls thedisplay device to display a recognition result corresponding to therecognized article. If it is determined to not recognize the article,extract a barcode from the image, the processor identifies the articlebased on the extracted barcode, and control the display device todisplay an identification result corresponding to the identifiedarticle. The processor performs a transaction settlement with respect tothe recognition result, if any, and the identification result, if any.

Descriptions of Overall Configuration of Check-Out System

Hereinafter, embodiments of an article reading device and a program willbe described with reference to drawings. In the present embodiment, asan example of the article reading device and the program, a check-outsystem 1 including an article reading device 101 provided in a storesuch as a supermarket is described.

FIG. 1 is an exterior perspective view illustrating an example of thecheck-out system 1. As illustrated in FIG. 1, the check-out system 1includes the article reading device 101 which captures an image of anarticle (or merchandise) which is a registration target when performingsales registration and reads feature data included in the captured imageof the article. The article reading device also reads identificationinformation of a barcode or the like attached to the article. Inaddition, the check-out system 1 includes a merchandise settlementdevice (POS (Point-of-Sale) terminal) 11 which performs salesregistration for merchandise read by the article reading device 101 andperforms a settlement process. The POS terminal 11 chooses a merchandisecandidate based on the feature data of the article output by the articlereading device 101. In addition, the POS terminal 11 provides themerchandise candidate to a salesperson (e.g., cashier) and allows thesalesperson to select the correct merchandise. In the check-out system 1of the embodiment, the article reading device 101 specifies merchandisewith respect to natural objects such as vegetables, fruits, meat, andthe like by object recognition. Other merchandise (for example,manufactured objects and the like) is specified by reading a barcodeattached to the merchandise.

In an example of FIG. 1, the POS terminal 11 is positioned on a drawer21, on a cash register counter 41. The POS terminal 11 has a keyboard 22on which keys for operation are arranged on an upper surface of the POSterminal 11. In addition, the POS terminal 11 has a first display device23 used for displaying registration information and receiving an inputof information required for settlement by the cashier. A second displaydevice 24 is positioned above the keyboard 22 and is used for displayingthe registration information to a customer. The first display device 23has a display panel 23 a such as a liquid crystal panel and is a touchinput type in which a touch panel 26 is overlaid on display panel 23 a.The second display device 24 has a display panel 24 a such as a liquidcrystal panel. The second display device 24 is positioned at an upperportion of the POS terminal 11 so as to be rotatable in a direction easyto see from the customer. The drawer 21 accommodates bills, coins, andthe like therein and is opened by an instruction from the POS terminal11.

Next to the cash register counter 41, a table 151 arranged is provided,forming an L-shape together with the cash register counter 41. On anupper surface of the table 151, a load receiving surface 152 for placinga shopping basket 153 and the like is formed. FIG. 1 illustrates anexample in which the shopping basket 153 includes a first shoppingbasket 153 a and a second shopping basket 153 b. The first shoppingbasket 153 a is a shopping basket which the customer brought, that is, ashopping basket containing articles (for example, an article G) beforereading. The second shopping basket 153 b is a shopping basket foraccommodating the article G after reading. After the cashier picks up,for example, the article G from the first shopping basket 153 a andcauses the article reading device 101 to read the article G, the cashiertransfers the article G to the second shopping basket 153 b. Theshopping basket 153 is not limited to a basket shape but may be a trayshape, a box shape, a bag shape, or the like.

The article reading device 101 is positioned at a central portion of theload receiving surface 152 of the table 151 and is connected with thePOS terminal 11 so as to be able to transmit and receive data by wire orwireless. The article reading device 101 includes a reading unit 110 inwhich a reading window 103 is disposed on a front surface (cashier side)in a housing 102. The reading window 103 is made of, for example, atransparent glass plate or the like with transparency. In addition, thearticle reading device 101 includes a display/operation unit 104 at anupper portion of the housing 102. The display/operation unit 104 has athird display device 106 such as a liquid crystal panel. The cashieruses the display/operation unit 104 for checking information of an imageand the like read by the reading unit 110 and for an input ofinformation when selecting merchandise among merchandise candidatesrecognized by the article reading device 101. A touch panel 105 isoverlaid on a screen of the third display device 106 so that a touchinput can be performed. In addition, a keyboard 107 is provided on aright side of the third display device 106, so that a keyboard input canbe performed by the cashier. Further, the article reading device 101includes a fourth display device 109 on a rear side (e.g., customerside) of the display/operation unit 104 for the customer to check areading result and the like of the article G by the customer.

The reading unit 110 includes an imaging unit 164 for imaging thearticle G in a back of the reading window 103. When the article G isimaged by the imaging unit 164, the reading unit 110 extracts featuredata of the article G among images of the imaged article G and outputsthe extracted feature data to the POS terminal 11. In addition, whenrecognizing that a code symbol such as a barcode for identifying thearticle G exists in the captured image, the reading unit 110 readscontents of the barcode and specifies the article G.

The display/operation unit 104 displays various screens such as apre-set screen, a selection screen of a merchandise candidate, and thelike output from the POS terminal 11 to the third display device 106.The pre-set screen includes a plurality of selection buttons forselecting merchandise. The selection screen is a screen on whichmerchandise candidate information read by the reading unit 110 isdisplayed. Details of a screen configuration will be described below.

In addition, the display/operation unit 104 displays an image capturedby the imaging unit 164 on the third display device 106 which is anexample of a display unit. By viewing the captured image displayed onthe third display device 106, the cashier checks whether or not thearticle G is properly captured. The cashier checks the captured imagedisplayed on the third display device 106. When an inappropriate imageis captured for reading such as occurrence of frame-out of the articleG, occurrence of a blur, or the like, the cashier instructs the readingunit 110 to read the article G again.

Descriptions of Hardware Configuration of Check-Out System

Next, a hardware configuration of the POS terminal 11 and the articlereading device 101 will be described. FIG. 2 is a block diagramillustrating the hardware configuration of the POS terminal 11 and thearticle reading device 101. The POS terminal 11 includes a CentralProcessing Unit (CPU) 61, a Read Only Memory (ROM) 62, a Random AccessMemory (RAM) 63, a Hard Disk Drive (HDD) 64, a connection interface 65,a printer 66, the drawer 21, the keyboard 22, the first display device23, the second display device 24, a communication interface 25, thetouch panel 26, and the like. The CPU 61, the ROM 62, and the RAM 63 areconnected with each other by an internal bus 27. In addition, all of thedrawer 21, the keyboard 22, the first display device 23, the seconddisplay device 24, the communication interface 25, the touch panel 26,the HDD 64, the connection interface 65, and the printer 66 areconnected to the internal bus 27 described above via variousinput/output circuits (not illustrated)

The CPU 61 is a central processing unit which controls the overall POSterminal 11. The ROM 62 is a nonvolatile memory which stores fixedprograms and the like. The RAM 63 is a volatile memory used by the CPU61 as a work area or the like.

The HDD 64 is a memory unit which stores various programs or variousfiles. The various programs include a program PR for merchandise salesdata processing including a process of determining a merchandisecandidate and displaying information indicating the merchandisecandidate on a selection screen, and the like. The various files includea PLU file F1 and the like that is, for example, transmitted from astore computer SC and stored. In addition, the HDD 64 stores aregistration table, a sales table, or the like for registeringmerchandise.

The communication interface 25 is configured with a network card or thelike for performing data communication with the store computer SC. Thestore computer SC is installed in a back office or the like of a store.The store computer SC may store the PLU file F1 and the like fortransmitting to the POS terminal 11 in an HDD (not illustrated).

The connection interface 65 is an interface for communicating with aconnection interface 175 or a connection interface 176 of the articlereading device 101. The communication is performed by wire or wireless.The printer 66 is a printing device which prints transaction contents ona receipt or the like and discharges the receipt.

The article reading device 101 includes the reading unit 110 and thedisplay/operation unit 104. The reading unit 110 has a CPU 161, a ROM162, a RAM 163, the imaging unit 164, an audio output unit 165, and theconnection interface 175. In addition, the display/operation unit 104has the connection interface 176, the touch panel 105, the third displaydevice 106, the keyboard 107, and the fourth display device 109.

The CPU 161, the ROM 162, and the RAM 163 are connected with each otherby an internal bus 166. In addition, all of the imaging unit 164, theaudio output unit 165, and the connection interface 175 are connected tothe internal bus 166 described above via various input/output circuits(not illustrated). Then, all of the touch panel 105, the third displaydevice 106, the keyboard 107, and the fourth display device 109 areconnected to the connection interface 176.

The CPU 161 is a central processing unit which controls the overallarticle reading device 101. The ROM 162 is a nonvolatile memory whichstores control programs and the like. The control program includes aprogram or the like for a process of extracting feature data from acaptured image and outputting the extracted feature data to the POSterminal 11. The RAM 163 is a volatile memory used by the CPU 161 as awork area or the like.

The imaging unit 164 is a color image sensor having an imaging elementsuch as a Charge Coupled Device (CCD), a Complementary Metal OxideSemiconductor (CMOS), or the like. The imaging unit 164 starts imagingby receiving an imaging start signal from the CPU 161, converts lightreflected from the article G held by the reading window 103 to anelectric signal at a predetermined frame rate (for example, 30 fps), andsequentially outputs the captured image.

The audio output unit 165 has a reproduction circuit, such as a speakerfor playing a pre-set reading sound (e.g., “beep”), warning sound,voice, and the like. The audio output unit 165 plays the reading sound,the warning sound, the voice, and the like based on a control signalfrom the CPU 161 and notifies a reading state or the like of the articleG by the sound.

Further, the connection interface 175 is connected to the CPU 161. Theconnection interface 175 communicates with the connection interface 65of the POS terminal 11 and thereby performs data transmission andreception between the article reading device 101 and the POS terminal11.

The connection interface 176 communicates with the connection interface65 of the POS terminal 11 and thereby performs data transmission andreception between the article reading device 101 and the POS terminal11.

Here, the PLU file F1 will be described. The PLU file F1 is a data filein which merchandise information and feature data of the merchandise areassociated with each other for each of the articles G. The merchandiseinformation is merchandise identification information (merchandise ID orthe like) for uniquely identifying merchandise, merchandiseclassification, an illustration image showing merchandise, informationindicating a unit price, and the like. The feature data includes, forexample, a surface state (e.g., texture) such as color tone orunevenness of the surface, which indicates a feature of each of thearticles G previously extracted from a sample image of each of thearticles G by a predetermined algorithm.

FIG. 3 illustrates an example of a data configuration of the PLU fileF1. The PLU file F1 stores data in which the merchandise information andthe feature data of the merchandise are associated with each other foreach of the articles G. In an example illustrated in FIG. 3, data suchas a merchandise ID field f1, a merchandise classification field f2, amerchandise name field f3, an unit price field f4, an illustration imagefield f5, and the like is stored as merchandise information. Data infeature data field f6 of the merchandise is associated with themerchandise information. The data stored in the feature data field f6 isused as data for verification when determining a similarity to bedescribed below. Data “XXXXXXXX” illustrated in the merchandise ID fieldf1 is a value uniquely corresponding to each piece of merchandise. Data“YYYYYYYY” illustrated in the feature data field f6 is a valuecalculated for each piece of merchandise.

Descriptions of Functional Configuration of Check-Out System

Next, a functional configuration of the POS terminal 11 and the articlereading device 101 of the check-out system 1 will be described. FIG. 4is a block diagram illustrating an example configuration of a functionalblock of each of the POS terminal 11 and the article reading device 101.In the article reading device 101, the CPU 161 illustrated in FIG. 2appropriately executes a program stored in the ROM 162. Accordingly, asillustrated in FIG. 4, an image capturing unit 51, a merchandisedetection unit 52, a barcode recognition unit 53, a feature dataextraction unit 54, and an object recognition application determinationunit 55 are realized as functional units by the CPU 161. In oneembodiment, the CPU 161 is a processor that is programmed to carry outthe functions of the image capturing unit 51, the merchandise detectionunit 52, the barcode recognition unit 53, the feature data extractionunit 54, and the object recognition application determination unit 55.In another embodiment, the CPU 61 is a hardware controller, e.g., anapplication specific integrated circuit (ASIC) and field programmablegate array (FPGA), that is configured to carry out the functions of theimage capturing unit 51, the merchandise detection unit 52, the barcoderecognition unit 53, the feature data extraction unit 54, and the objectrecognition application determination unit 55.

In addition, in the POS terminal 11, the CPU 61 illustrated in FIG. 2appropriately executes a program stored in the ROM 62 and the program PRstored in the HDD 64. Accordingly, as illustrated in FIG. 4, amerchandise candidate extraction unit 71, a display control unit 72, aninput receiving unit 73, and a sales registration unit 74 are realizedas functional units by the CPU 61. In one embodiment, the CPU 61 is aprocessor that is programmed to carryout the functions of themerchandise candidate extraction unit 71, the display control unit 72,the input receiving unit 73, and the sales registration unit 74. Inanother embodiment, the CPU 61 is a hardware controller, e.g., anapplication specific integrated circuit (ASIC) and field programmablegate array (FPGA), that is configured to carry out the functions of themerchandise candidate extraction unit 71, the display control unit 72,the input receiving unit 73, and the sales registration unit 74.

The image capturing unit 51 outputs an imaging start signal to theimaging unit 164 (FIG. 2) and starts an imaging operation of the imagingunit 164. After that, the image capturing unit 51 obtains the imagecaptured and output by the imaging unit 164 and sequentially stores theimage in the RAM 163 (FIG. 2).

The merchandise detection unit 52 detects that the article G is heldnear the reading window 103 (FIG. 1) based on a plurality of successiveimages captured and output by the imaging unit 164. Specifically, themerchandise detection unit 52 reads the captured image output by theimaging unit 164 (FIG. 2) from the RAM 163 (FIG. 2) and comparesluminance of the captured image with luminance of a captured image readmost recently. The merchandise detection unit 52 determines that thecaptured image includes reflection of hands, the article G, and the likewith a condition that luminance gradation is changed according tocomparison of luminance. Further, the merchandise detection unit 52binarizes the captured image having the reflection and the successivecaptured image and extracts a contour line of the article G. When aposition of a contour line of the successive captured image is changed,the merchandise detection unit 52 specifies that an area inside thecontour line is an area indicating the article G held by the readingwindow 103. In this way, the merchandise detection unit 52 detects thatthe article G is held by the reading window 103.

A method of specifying that the article G is held is not limited to themethod described above and other methods may be used. For example, itmay be specified not by the contour line of the article G in thecaptured image but by presence or absence of a flesh color area. Thatis, in a case where there is the flesh color area inside the capturedimage, it is expected that the cashier's hands are reflected. Therefore,for example, a contour line may be extracted from the captured image andan area inside the contour line of a position at which the article isgripped may be specified as an area indicating the article G amongcontour lines showing a shape of the hands.

The barcode recognition unit 53 is an example of a barcode reading unitand recognizes that a code symbol such as a barcode is imaged in thecaptured image. In addition, the barcode recognition unit 53 performs areading process of reading a recognized barcode and provides a readingresult to the merchandise candidate extraction unit 71. In addition, ina case where the code symbol such as a barcode or the like is not imagedin the captured image, the barcode recognition unit 53 providesnotification that the barcode is not read to the merchandise candidateextraction unit 71. Since a method of determining whether or not thebarcode is imaged in the captured image or a method of reading codeinformation of the barcode imaged in the captured image is already usedas a known technique, details of the description will be omitted.

The feature data extraction unit 54 is an example of an extraction unitand extracts feature data 80 a included in a recognition target articlefor object recognition from an area of the article G imaged in thecaptured image. Then, the feature data extraction unit 54 outputs theextracted feature data 80 a to the merchandise candidate extraction unit71 of the POS terminal 11. Specifically, the feature data extractionunit 54 reads a captured image from the RAM 163. Then, the feature dataextraction unit 54 specifies an area of the article G from the capturedimage. Next, the feature data extraction unit 54 extracts the featuredata 80 a corresponding to the feature data field f6 registered in thePLU file F1 in advance as the feature data 80 a included in therecognition target article for object recognition with respect to aninside of the specified area. The feature data extraction unit 54outputs the extracted feature data 80 a to the connection interface 175(FIG. 2). The connection interface 175 outputs the received feature data80 a to the connection interface 65 (FIG. 2) and provides the featuredata 80 a via the connection interface 65 to the merchandise candidateextraction unit 71.

In addition, the feature data extraction unit 54 extracts feature data80 b for determining whether or not the article G is a non-targetarticle for object recognition from an area of the article G imaged inthe captured image. For example, in the present example, merchandise isidentified by object recognition for natural objects such as vegetables,fruits, meat, and the like and the merchandise is identified by abarcode attached to merchandise for manufactured objects. Then, thefeature data extraction unit 54 extracts the feature data 80 b fordistinguishing whether an area of the article G imaged in the capturedimage is manufactured objects or natural objects. Specifically, thefeature data 80 b which is not present in natural objects such asvegetables, fruits, meat, and the like among feature data common toartificial objects such as plastic, paper, cloth, or the like isselected and is used.

The object recognition application determination unit 55 is an exampleof a determination unit and determines whether or not the article Gimaged in the captured image is a non-target article for objectrecognition based on whether the feature data 80 b is extracted by thefeature data extraction unit 54. Further, the object recognitionapplication determination unit 55 determines that it is necessary toread a barcode so as to identify the article G on the condition that thearticle G imaged in the captured image is determined to be amanufactured object, that is, a non-target article for objectrecognition. On the other hand, the object recognition applicationdetermination unit 55 determines that it is necessary to perform objectrecognition so as to identify the article G on condition that thearticle G imaged in the captured image is determined to be a naturalobject, that is, a recognition target article for object recognition. Adetermination result of the object recognition application determinationunit 55 is provided to the merchandise candidate extraction unit 71 andthe display control unit 72.

The merchandise candidate extraction unit 71 is an example of an objectrecognition unit and performs an object recognition process forrecognizing the article G inside the captured image. Specifically, themerchandise candidate extraction unit 71 calculates a similarity bycomparing the feature data 80 a of the article G provided by the featuredata extraction unit 54 with the feature data field f6 of each piece ofmerchandise registered in the PLU file F1 (FIG. 3). Further, themerchandise candidate extraction unit 71 chooses merchandise havingsimilarity is equal to or larger than a predetermined threshold amongthe PLU file F1 as a candidate of the article G. The merchandisecandidate extraction unit 71 outputs the chosen merchandise candidateinformation to the display control unit 72. In addition, the merchandisecandidate extraction unit 71 identifies the merchandise by verifying themerchandise ID field f1 read by the barcode recognition unit 53 with thePLU file F1 with respect to the article G to which a barcode isattached. The merchandise candidate extraction unit 71 outputs theidentifies merchandise information to the display control unit 72.

The object recognition for recognizing the article G included in animage is also called generic object recognition. Since variousrecognition techniques used for generic object recognition are describedin detail in the related art, the description will be omitted. Examplegeneric object recognition techniques are described in the following:Yanai Keiji, “Current and Future of Generic Object Recognition”, Journalof Information Processing Society, Vol. 48, No. SIG16 [Search on Aug. 5,2016], Internet <URL: http://mm.cs.uec.ac.jp/IPSJ-TCVIM-Yanai.pdf>

Jamie Shotton, “Semantic Texton Forests for Image Categorization andSegmentation”, [Search on Aug. 5, 2016], Internet <URL:http://mieng.cam.acuk/˜cipolla/publications/inproceedings/2008-CVPR-semantic-texton-forests.pdf>

Here, in a case where, for example, the feature data field f6 of each ofarticles inside the PLU file F1 is set to 100% “similarity: 1.0”, thesimilarity calculated when the merchandise candidate extraction unit 71performs object recognition and specifies the article G indicates howsimilar the feature data 80 a of the article G imaged in the capturedimage is. A similarity may be calculated by absolute evaluation orrelative evaluation.

For example, a similarity is calculated by absolute evaluation. In thiscase, the feature data 80 a of the article G imaged in the capturedimage and the feature data field f6 of each of articles registered inthe PLU file F1 are compared one by one and a similarity calculated as aresult of this comparison is adopted as-is. Here, it is assumed that asimilarity of 95% is set as a threshold. In this case, the merchandisecandidate extraction unit 71 sequentially assigns merchandise having asimilarity of 95% equal to or more in order of similarities and outputsthe merchandise to the display control unit 72.

In addition, similarities may be calculated by relative evaluation. Inthis case, similarities are calculated so that a total of thesimilarities to the merchandise of the PLU file F1 is 1.0 (100%). Forexample, it is assumed that the feature data of five pieces ofmerchandise in the PLU file F1 are registered in the feature data fieldf6. In this case, for example, a similarity for the first merchandise is“0.6”, a similarity for the second merchandise is “0.1”, a similarityfor the third merchandise is “0.1”, a similarity for the fourthmerchandise is “0.1”, and a similarity for the fifth merchandise is“0.1”, so that a total of the similarities is always 1.0. Here, it isassumed that a similarity 0.5 is a threshold. In this case, themerchandise candidate extraction unit 71 sequentially assignsmerchandise information of merchandise having a similarity 0.5 equal toor more in order of similarities and outputs the merchandise to thedisplay control unit 72.

The display control unit 72 is an example of a display control unit andcontrols respective display states of the first display device 23, thesecond display device 24, the third display device 106, and the fourthdisplay device 109 (all are illustrated in FIG. 2). The display controlunit 72 controls the display states of the first display device 23 andthe second display device 24 via the internal bus 27. In addition, thedisplay control unit 72 controls the display states of the third displaydevice 106 and the fourth display device 109 via the connectioninterface 65 and the connection interface 176. Hereinafter, aconfiguration will be described with respect to exemplary embodiments ofthe present disclosure, in which the display control unit 72 controlsthe display state of the third display device 106.

The display control unit 72 displays a pre-set screen G1 (see FIG. 5A)on which a plurality of selection buttons for selecting merchandise arearranged at predetermined positions on the third display device 106.Further, when receiving merchandise candidate information from themerchandise candidate extraction unit 71, the display control unit 72displays a selection screen G2 (see FIG. 5B) having merchandisecandidate information with respect to the third display device 106.

Specifically, the display control unit 72 generates update informationof the pre-set screen G1 as a setting indicating a merchandise candidateby setting a display of selection buttons of, for example, top fourmerchandise candidates chosen as merchandise candidate information. Thedisplay control unit 72 outputs update information of the pre-set screenG1 to the connection interface 65. The output update information isoutput from the connection interface 65 to the third display device 106via the connection interface 176. Accordingly, a display screen of thethird display device 106 transitions from the pre-set screen G1 to themerchandise candidate selection screen G2.

In addition, on condition that the object recognition applicationdetermination unit 55 determines that it is necessary to perform anobject recognition process for specifying the article G imaged in thecaptured image, the display control unit 72 controls a display form whena captured image captured by the imaging unit 164 is displayed on thethird display device 106. Specific control contents of the display formwill be described below.

The input receiving unit 73 receives various input operations from aninput/output device such as the keyboards 22 and 107 and the touchpanels 26 and 105. For example, while a merchandise candidate selectionscreen is displayed on the third display device 106, the cashier pressesa selection button of the keyboard 107 or touches a selection button ofthe touch panel 105. In this case, operation information of the operatedselection button is output from the input/output device to theconnection interface 176. The operation information is output from theconnection interface 176 to the connection interface 65 and is providedto the input receiving unit 73 via the connection interface 65.

Based on the operation information of the selection button received fromthe keyboard 107 or the touch panel 105 by the input receiving unit 73,the sales registration unit sales-registers merchandise information of apiece of merchandise corresponding to the operation information. Thatis, the sales registration unit adds the merchandise information to aregistration table or the like. In addition, based on the closingoperation information received from the touch panel 26 by the inputreceiving unit 73, the sales registration unit 74 performs a settlementprocess of one transaction of all merchandise that has beensales-registered and registers the sales information to a sales table orthe like.

Descriptions of Contents of Display Screen

Next, contents of a screen displayed on the third display device 106 bythe check-out system 1 when registering the article G will be described.FIGS. 5A to 5D illustrate examples of screens displayed on the thirddisplay device 106. FIG. 5A is an example of the pre-set screen G1displayed when the check-out system 1 starts to operate. FIG. 5B is anexample of the selection screen G2 displayed when the merchandisecandidate extraction unit 71 extracts merchandise candidates from imagesof the article G captured by the imaging unit 164. FIG. 5C is an exampleof a reading result screen G3 displayed when the barcode recognitionunit 53 reads a barcode attached to the article G. FIG. 5D is an exampleof a re-recognition selection screen G4 displayed when the merchandisecandidate extraction unit 71 cannot extract the merchandise candidatesfrom images of the article G captured by the imaging unit 164.

As illustrated in FIG. 5A, the pre-set screen G1 has tabs T1, T2, . . ., a selection button display area E1, and the like. The tabs T1, T2, . .. are buttons for switching selection buttons by categories anddisplaying the selection buttons. The selection button display area E1is a display area in which selection buttons K1, K2, K3, K4, K5, . . .of a category designated by the tabs T1, T2, . . . are displayed atrespective positions determined.

In a case of performing sales registration of merchandise withoutperforming the object recognition process or the barcode reading processdescribed above, the cashier designates a category among the tabs T1,T2, . . . by a touch input of the screen and further selects thecorresponding selection buttons K1, K2, K3, K4, K5, . . . from theselection button display area E1 corresponding to the selected category.Operation information of the selection buttons K1, K2, K3, K4, K5, . . .selected by this operation is received by the input receiving unit 73 ofthe POS terminal 11. Merchandise information corresponding to theoperated selection buttons K1, K2, K3, K4, K5, . . . is retrieved fromthe PLU file F1 (FIG. 2) and is sales-registered by the salesregistration unit 74 (FIG. 4).

In a case where the barcode recognition unit 53 (FIG. 4) performs abarcode reading process of reading a barcode of the article G andrecognizes contents of the barcode, the display control unit 72 displaysthe reading result screen G3 illustrated in FIG. 5C. That is, thedisplay control unit 72 displays only a selection button (in this case,the selection button K5) corresponding to the contents of the recognizedbarcode in the selection button display area E1 of the reading resultscreen G3 in an operable state. The display control unit 72 displaysselection buttons other than the selection button K5 in an inoperablestate, such as greyed-out on the reading result screen G3. After thecashier checks that the selection button K5 displayed in the operablestate corresponds to the article G, the cashier presses the selectionbutton K5. Then, operation information of the selection button K5 isreceived by the input receiving unit 73 of the POS terminal 11.Merchandise information corresponding to the operated selection buttonK5 is called from the PLU file F1 (FIG. 2) and is sales-registered bythe sales registration unit 74 (FIG. 4). Here, a configuration in whicha result of the barcode reading process is displayed as the readingresult screen G3 and the cashier performs sales registration afterchecking the displayed contents is described, the configuration is notlimited thereto. That is, on condition that the article G is specifiedbased on the result of the barcode reading process, sales registrationmay be performed without the cashier re-checking a result of specifyingthe article G.

Further, as illustrated in FIG. 5B, in a case where it is necessary toperform object recognition for specifying the article G imaged in thecaptured image, the display control unit 72 displays the selectionscreen G2 having merchandise candidates obtained by the objectrecognition process. At this time, the display control unit 72simultaneously displays an image of the article G captured by theimaging unit 164 in a captured image display area E2. At this time, thedisplay control unit 72 superimposes and displays an object detectionframe W1 indicating a position of the article G object-recognized by themerchandise candidate extraction unit 71 on the image of the article G.The cashier sees the displayed image and checks that the article G isproperly imaged in the captured image display area E2 and that theobject detection frame W1 is correctly superimposed on the article G.

The display control unit 72 displays, for example, the top four piecesof merchandise with the highest similarity among merchandise candidatesextracted by the merchandise candidate extraction unit 71 (FIG. 4) in amerchandise candidate display area E3 on the selection screen G2. Thatis, the display control unit 72 displays selection buttons (K10, K11,K12, and K13) in the merchandise candidate display area E3.

In addition, on the condition that the article G imaged in the capturedimage is a target article for object recognition and the merchandisecandidate extraction unit 71 cannot extract a merchandise candidate fromthe captured image, the display control unit 72 displays there-recognition selection screen G4 suggesting execution ofre-recognition illustrated in FIG. 5D. That is, the display control unit72 displays a message suggesting execution of re-recognition and are-recognition start button K20 in the captured image display area E2 ofthe re-recognition selection screen G4. Further, the display controlunit 72 displays the merchandise candidate display area E3 of there-recognition selection screen G4 in an inoperable state such asgreyed-out.

Descriptions of Screen Transition in Third Display Device

Next, screen transition displayed on the third display device 106 by thecheck-out system 1 will be described. FIG. 6 is a state transitiondiagram illustrating an example of screen transitions on the thirddisplay device 106.

In a state (node N1) where the pre-set screen G1 (FIG. 5A) is displayedon the third display device 106, when the object recognition applicationdetermination unit 55 determines that the article G is a target articlefor object recognition, the merchandise candidate extraction unit 71performs the object recognition process. Then, when a merchandisecandidate is extracted (arc A1), the display control unit 72 displaysthe selection screen G2 having the captured image and the merchandisecandidate on the third display device 106 (node N2).

On the other hand, in a state of the node N1, when the merchandisecandidate extraction unit 71 performs the object recognition process andthe merchandise candidate is not extracted (arc A2), the display controlunit 72 displays the re-recognition selection screen G4 on the thirddisplay device 106 (node N4).

Further, in a state of the node N1, when the object recognitionapplication determination unit 55 determines that the article G is anon-target article for object recognition, the barcode recognition unit53 performs a barcode reading process (arc A3). Then, the displaycontrol unit 72 displays the reading result screen G3 on the thirddisplay device 106 (node N3).

In a state (node N2) where the selection screen G2 is displayed on thethird display device 106, when the cashier presses the selection buttons(K10, K11, K12, and K13) (FIG. 5B) corresponding to the article G in theselection screen G2 and performs a registration operation (arc A4), thesales registration unit 74 (FIG. 4) registers the article G as purchasedmerchandise. Then, the display control unit 72 displays the pre-setscreen G1 on the third display device 106 (node N1).

In a state (node N4) where the re-recognition selection screen G4 isdisplayed on the third display device 106, when the cashier presses there-recognition start button K20 (FIG. 5D), the article reading device101 performs the object recognition process again. As a result, when amerchandise candidate is extracted (arc A5), the display control unit 72displays the selection screen G2 having the captured image and themerchandise candidate on the third display device 106 (node N2).

On the other hand, in a state of the node N4, when the merchandisecandidate extraction unit 71 performs the object recognition processagain and the merchandise candidate is not extracted (arc A6), thedisplay control unit 72 displays the re-recognition selection screen G4on the third display device 106 again (node N4).

In a state (node N3) where the reading result screen G3 is displayed onthe third display device 106, when the cashier presses the selectionbutton K5 (FIG. 5C) selected by the barcode reading process and performsthe registration operation on the reading result screen G3 (arc A7), thesales registration unit 74 registers the article G as purchasedmerchandise. Then, the display control unit 72 displays the pre-setscreen G1 on the third display device 106 (node N1).

Descriptions of Operation of Check-Out System

Next, an operation example of the check-out system 1 will be described.FIG. 7 is a sequence diagram illustrating a process example forperforming one transaction of merchandise registration and a settlementprocess of the check-out system 1. In the following description, in thePOS terminal 11, it is assumed that the program stored in the ROM 62 andthe program PR stored in the HDD 64 are executed by the CPU 61 andfunctional units such as the merchandise candidate extraction unit 71,the display control unit 72, the input receiving unit 73, the salesregistration unit 74, and the like are realized. In addition, in thearticle reading device 101, it is assumed that the program stored in theROM 162 is executed by the CPU 161 and functional units such as theimage capturing unit 51, the merchandise detection unit 52, the barcoderecognition unit 53, the feature data extraction unit 54, the objectrecognition application determination unit 55, and the like arerealized. Further, in the article reading device 101, the pre-set screenG1 (FIG. 5A) is displayed on the third display device 106 and theimaging unit 164 is also always activated.

First, the article reading device 101 detects that the article G is heldby the reading window 103 and performs a reading process of the articleG (step S1). Specifically, when the cashier picks up the article Gbefore reading from the first shopping basket 153 a illustrated in FIG.1 and moves the article G to the second shopping basket 153 b, thearticle G is held by the reading window 103 and an appearance of themerchandise is read by the imaging unit 164 (FIG. 2). The articlereading device 101 obtains an image captured by the imaging unit 164from the image capturing unit 51 (FIG. 4) and stores the image in theRAM 163. Then, the merchandise detection unit 52 detects the article Gfrom the image. At this time, the feature data extraction unit 54extracts the feature data 80 b of the detected article G and the objectrecognition application determination unit 55 determines whether or notthe article G is a non-target article for object recognition based onthe feature data 80 b.

Next, the CPU 161 of the article reading device 101 provides the featuredata 80 a extracted by the feature data extraction unit 54 to themerchandise candidate extraction unit 71 of the POS terminal 11 (stepS2). Further, based on a determination result of the object recognitionapplication determination unit 55, on condition that the article G isnot a non-target article for object recognition, the CPU 161 providesinstructions to perform the object recognition process to themerchandise candidate extraction unit 71.

In the POS terminal 11, the merchandise candidate extraction unit 71performs the object recognition process and extracts merchandiseinformation from the PLU file F1 in descending order of similaritiesbased on the feature data 80 a notified from the article reading device101. The merchandise candidate extraction unit 71 outputs the extractedmerchandise candidate information to the display control unit 72 (stepS3). The number of pieces of output merchandise candidate informationmay be appropriately set. In the present embodiment, the top four piecesof merchandise candidate information are extracted.

Further, in the POS terminal 11, the display control unit 72 outputs theselection buttons (K10, K11, K12, and K13) (FIG. 5B) corresponding totop four merchandise candidates extracted as merchandise candidateinformation to the third display device 106 of the article readingdevice 101 as update information of the pre-set screen G1 (step S4).

Based on the update information output by the display control unit 72,the article reading device 101 displays the selection screen G2 (FIG.5B) of the merchandise candidates to the third display device 106 (stepS5).

The cashier selects a selection button (any one of K10, K11, K12, andK13 illustrated in FIG. 5B) corresponding to the merchandise candidatesof the selection screen G2 via the touch panel 105 (or the keyboard 107)which is the input receiving unit 73. Then, in the article readingdevice 101, an input/output circuit (not illustrated) of the touch panel105 (or the keyboard 107) notifies operation information of the articlereading device 101 to the input receiving unit 73 of the POS terminal 11(step S6).

When the input receiving unit 73 of the POS terminal 11 receivesmerchandise information corresponding to the article G selected by thecashier, the sales registration unit 74 registers the merchandiseinformation to a registration table or the like (step S7).

The processes from step S1 to step S7 described above are performed forall of the articles G in the first shopping basket 153 a illustrated inFIG. 1. After that, the cashier presses a sum key of the keyboard 22 orthe touch panel 26 in the POS terminal 11. The input receiving unit 73of the POS terminal 11 receives a notification that a pressing operationis performed, and the sales registration unit 74 performs a merchandisesettlement process and registers sales information to a sales table orthe like (step S8).

Descriptions of Flow of Process Performed by Check-Out System

Next, a flow of processes performed by the check-out system 1 will bedescribed. FIG. 8 is a flowchart illustrating an example sequence ofoperations for a series of merchandise registration processes performedby the check-out system 1.

The display control unit 72 displays the pre-set screen G1 (FIG. 5A) onwhich predetermined selection buttons (K1, K2, . . . ) are displayed onthe third display device 106 (step S10).

The image capturing unit 51 performs an imaging process of capturing animage of the article G by controlling the imaging unit 164 (FIG. 2)(step S12).

The merchandise detection unit 52 determines whether or not merchandiseis detected (step S14). When the merchandise detection unit 52 detectsthe merchandise (step S14: Yes), the process proceeds to step S16.Otherwise (step S14: No), the process returns to step S12.

The object recognition application determination unit 55 determineswhether or not the article G is a non-target article for objectrecognition based on the feature data 80 b of the article G imaged inthe captured image (step S16). When the article G is a non-targetarticle for object recognition (step S16: Yes), the process proceeds tostep S18. Otherwise (step S16: No), the process proceeds to step S30.

The barcode recognition unit 53 performs a barcode reading process ofreading contents of a barcode with respect to the captured image (stepS18).

The barcode recognition unit 53 determines whether or not the contentsof the barcode are read (step S20). When the contents of the barcode areread (step S20: Yes), the process proceeds to step S22. Otherwise (stepS20: No), the process proceeds to step S26.

The display control unit 72 displays the reading result screen G3 (FIG.5C) on the third display device 106 (step S22).

The sales registration unit 74 receives a notification that the cashierperforms an operation of selecting merchandise read by the barcodereading process with respect to the input receiving unit 73, andperforms a merchandise registration process (step S24). After that, theprocess in FIG. 8 is ended. Although not described in the flowchart ofFIG. 8, in a case where unregistered merchandise remains, the processreturns to step S10 and the same process is repeated. In addition, asdescribed above, when the barcode recognition unit 53 reads the contentsof the barcode, a merchandise registration process may be performedwithout displaying the reading result screen G3.

In step S20, when it is determined that the barcode recognition unit 53does not read the contents of the barcode (step S20: No), the barcoderecognition unit 53 determines whether or not a predetermined timeelapses (step S26). When the predetermined time elapses (step S26: Yes),the process proceeds to step S28. Otherwise (step S26: No), the processreturns to step S18.

When the barcode recognition unit 53 does not read the barcode evenafter the predetermined time elapses (step S26: Yes), the displaycontrol unit 72 displays guidance such as “Please change the directionto hold the merchandise” or the like on the third display device 106(step S28). After that, the process returns to step S18.

In step S16, when the object recognition application determination unit55 determines that the article G imaged in the captured image is not anon-target article for object recognition (step S16: No), themerchandise candidate extraction unit 71 performs the object recognitionprocess (step S30).

The display control unit 72 displays the captured image on the thirddisplay device 106 (step S32).

As a result of performing the object recognition process, themerchandise candidate extraction unit 71 determines whether or not amerchandise candidate is obtained (step S34). When the merchandisecandidate is obtained (step S34: Yes), the process proceeds to step S36.When the merchandise candidate is not obtained (step S34: No), theprocess proceeds to step S40.

The display control unit 72 displays the selection screen G2 (FIG. 5B)having top four merchandise candidates in descending order ofsimilarities on the third display device 106 (step S36).

The input receiving unit 73 determines whether or not merchandise isselected from the selection screen G2 (step S38). When the merchandiseis selected (step S38: Yes), the process proceeds to step S24. Otherwise(step S38: No), the process returns to step S36.

In step S34, when the merchandise candidate is not obtained (step S34:No), the display control unit 72 displays the re-recognition selectionscreen G4 (FIG. 5D) suggesting execution of re-recognition on the thirddisplay device 106 (step S40).

The input receiving unit 73 determines whether or not re-recognition isinstructed (step S42). When the re-recognition is instructed (step S42:Yes), the process proceeds to step S30. Otherwise (step S42: No), theprocess returns to step S40.

In the present embodiment, on the third display device 106, the displaycontrol unit 72 displays captured images which have not yet beendisplayed at a timing during which object recognition is executed andremoves an image of the article G at a timing during which themerchandise candidate is selected. Display control performed by thedisplay control unit 72 is not limited thereto. That is, the displaycontrol unit 72 may always display the captured image captured by theimaging unit 164 in a display screen of the third display device 106with a predetermined size and may perform display control to enlarge adisplay size of the captured image at a timing during which objectrecognition is executed. At this time, the display control unit 72returns the captured image enlarged to a predetermined size at a timingduring which the cashier selects a merchandise candidate.

Descriptions of Flow of Modification Example of Process Performed byCheck-Out System

Next, a modification example of a series of processes performed by thecheck-out system 1 will be described with reference to FIG. 9. Since aconfiguration of the modification example is the same as that of thecheck-out system. 1 described above, each of components will bedescribed using the same reference numerals.

In this modification example, an operation of the object recognitionapplication determination unit 55 is different from the configurationdescribed above. That is, in the configuration (FIG. 8) described above,the object recognition application determination unit 55 determineswhether or not the article G is a non-target article for objectrecognition based on the feature data 80 b of the article G imaged inthe captured image. However, in the modification example (FIG. 9)described below, the object recognition application determination unit55 determines whether or not the article G is a non-target article forobject recognition based on a barcode reading result from the article G.

FIG. 9 is a flow chart illustrating the modification example of a flowof a series of processes performed by the check-out system 1. Thedisplay control unit 72 displays the pre-set screen G1 (FIG. 5A) onwhich predetermined selection buttons (K1, K2, . . . ) are disposed onthe third display device 106 (step S50).

The image capturing unit 51 performs an imaging process of capturing animage of the article G by controlling the imaging unit 164 (FIG. 2)(step S52).

The merchandise detection unit 52 determines whether or not merchandiseis detected (step S54). When the merchandise detection unit 52 detectsthe merchandise (step S54: Yes), the process proceeds to step S56.Otherwise (step S54: No), the process returns to step S52.

The barcode recognition unit 53 performs a barcode reading process ofreading contents of a barcode with respect to the captured image (stepS56).

The barcode recognition unit 53 determines whether or not the contentsof the barcode are read (step S58). When the contents of the barcode areread (step S58: Yes), the process proceeds to step S60. Otherwise (stepS58: No), the process proceeds to step S64.

The display control unit 72 displays the reading result screen G3 (FIG.5C) on the third display device 106 (step S60).

The sales registration unit 74 receives notification that the cashierperforms an operation of selecting merchandise read by the barcodereading process with respect to the input receiving unit 73, andperforms a merchandise registration process (step S62). After that, theprocess in FIG. 9 is ended. Although not described in a flowchart ofFIG. 9, in a case where unregistered merchandise remains, the processreturns to step S50 and the same process is repeated. In addition, whenthe barcode recognition unit 53 reads the contents of the barcode, amerchandise registration process may be performed without displaying thereading result screen G3.

In step S58, when it is determined that the barcode recognition unit 53does not read the contents of the barcode (step S58: No), the barcoderecognition unit 53 determines whether or not a predetermined timeelapses (step S64). When the predetermined time elapses (step S64: Yes),the process proceeds to step S66. When the predetermined time does notelapse (step S64: No), the process returns to step S56.

In step S64, when it is determined that the barcode recognition unit 53does not read the barcode even after the predetermined time elapses(step S64: Yes), the object recognition application determination unit55 determines that the article G imaged in the captured image is atarget article for object recognition and the merchandise candidateextraction unit 71 performs the object recognition process (step S66).

The display control unit 72 displays the captured image on the thirddisplay device 106 (step S68).

As a result of performing the object recognition process, themerchandise candidate extraction unit 71 determines whether or not amerchandise candidate is obtained (step S70). When the merchandisecandidate is obtained (step S70: Yes), the process proceeds to step S72.When the merchandise candidate is not obtained (step S70: No), theprocess proceeds to step S76.

The display control unit 72 displays the selection screen G2 (FIG. 5B)having top four merchandise candidates in descending order ofsimilarities on the third display device 106 (step S72).

The input receiving unit 73 determines whether or not merchandise isselected from the selection screen G2 (step S74). When the merchandiseis selected (step S74: Yes), the process proceeds to step S62. Otherwise(step S74: No), the process returns to step S72.

In step S70, when the merchandise candidate is not obtained (step S70:No), the display control unit 72 displays the re-recognition selectionscreen G4 (FIG. 5D) suggesting execution of re-recognition on the thirddisplay device 106 (step S76).

The input receiving unit 73 determines whether or not re-recognition isinstructed (step S78). When the re-recognition is instructed (step S78:Yes), the process returns to step S66. Otherwise (step S78: No), theprocess returns to step S76.

As described above, in the check-out system 1 according to theembodiment, based on the feature data 80 b of the article G extracted bythe feature data extraction unit 54, the object recognition applicationdetermination unit 55 determines whether the article G is a targetarticle requiring a recognition result of the merchandise candidateextraction unit 71 or a target article requiring a reading result of thebarcode recognition unit 53 among captured images obtained by imaging anappearance of the article G. Based on a determination result of theobject recognition application determination unit 55, the displaycontrol unit 72 controls a display state of the captured image for thethird display device 106. Therefore, in a case where the article G is atarget article requiring a recognition result of the merchandisecandidate extraction unit 71, it is possible to display the capturedimage of the article G for easy viewing. Thereby, when an image of thearticle G is captured, it is possible to prevent the article G fromprotruding from an imaging range of merchandise and to capture the imagefor easy recognizing.

In addition, in the check-out system 1 according to the embodiment,based on the feature data 80 b, extracted by the feature data extractionunit 54, representing a similarity with a non-target article for objectrecognition, the object recognition application determination unit 55distinguishes whether or not the article G is a non-target article forobject recognition. Therefore, it is possible to reliably determinewhether or not object recognition is necessary when registering thearticle G.

In the check-out system 1 according to the embodiment, on condition thata code symbol cannot be read within a predetermined time after thebarcode recognition unit 53 starts to read the code symbol of thearticle G, the object recognition application determination unit 55determines that the article G is a target article for objectrecognition. Therefore, it is possible to reliably determine whether ornot object recognition is necessary when registering the article G.

Further, in the check-out system 1 according to the embodiment, oncondition that the object recognition application determination unit 55determines that the article G is a target article for objectrecognition, the display control unit 72 displays the captured image ofthe article G on the third display device 106. Then, on condition thatit is determined that the article G is a non-target article for objectrecognition, the object recognition application determination unit 55does not display the captured image of the article G on the thirddisplay device 106. Therefore, when the object recognition process isnecessary, it is possible to display the captured image of the article Gfor easy viewing without performing a special operation.

In addition, in the check-out system 1 according to the embodiment, oncondition that the object recognition application determination unit 55determines that the article G is a non-target article for objectrecognition, the display control unit 72 displays the captured image onthe third display device 106 at a predetermined display size. Then, oncondition that it is determined that the article G is a target articlefor object recognition, the object recognition application determinationunit 55 enlarges the captured image more than the predetermined displaysize and displays the captured image on the third display device 106.Therefore, when the object recognition process is necessary, it ispossible to display the captured image of the merchandise for easyviewing without performing a special operation.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel embodiments described hereinmay be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the embodimentsdescribed herein may be made without departing from the spirit of theinventions. The accompanying claims and their equivalents are intendedto cover such forms or modifications as would fall within the scope andspirit of the inventions.

What is claimed is:
 1. An article reading device for registering anarticle for purchase, the article reading device comprising: a displaydevice; an image capturing device configured to continuously captureimages of an object within an imaging range; and a processor configuredto: perform first object recognition on one or more first imagescaptured by the capturing device and attempt to read a code symbol fromthe first images, when a code symbol is read from the first images,acquire article information of an article corresponding to the codesymbol, and control the display device to display a first screen showinga button for selecting the article for registration using the articleinformation, and when a code symbol is not read from the first images,perform second object recognition on one or more second images capturedby the capturing device to identify an article for registration andcontrol the display device to display a second screen showing the secondimages.
 2. The article reading device according to claim 1, wherein thefirst screen further shows the first images.
 3. The article readingdevice according to claim 1, wherein the second images are enlarged inthe second screen.
 4. The article reading device according to claim 1,wherein the processor is further configured to, when performing thesecond object recognition, determine one or more candidate articles, andthe candidate articles are also displayed in the second screen.
 5. Thearticle reading device according to claim 4, wherein one of thecandidate articles is selectable for registration through the secondscreen.
 6. The article reading device according to claim 4, furthercomprising: a network interface, wherein the processor is configured tocontrol the network interface to transmit a result of the second objectrecognition to a point-of-sale (POS) terminal.
 7. The article readingdevice according to claim 4, wherein the processor is further configuredto, when no candidate article is determined, control the display deviceto display a third screen indicating a button to retry the second objectrecognition.
 8. The article reading device according to claim 1, whereinthe processor is further configured to, before performing the firstobject recognition, control the display device to display a presetscreen showing a plurality of buttons for selecting an article forregistration.
 9. The article reading device according to claim 8,wherein the processor is configured to control the display device todisplay the first screen in place of the preset screen when the codesymbol is read.
 10. The article reading device according to claim 9,wherein the processor is configured to control the display device todisplay the preset screen after the button in the first screen isselected.
 11. A method performed by an article reading device forregistering an article for purchase, the method comprising: performingfirst object recognition on one or more first images of an objectcaptured by a capturing device and attempting to read a code symbol fromthe first images, when a code symbol is read from the first images,acquiring article information of an article corresponding to the codesymbol, and displaying a first screen showing a button for selecting thearticle for registration using the article information, and when a codesymbol is not read from the first images, performing second objectrecognition on one or more second images captured by the capturingdevice to identify an article for registration and displaying a secondscreen showing the second images.
 12. The method according to claim 11,wherein the first screen further shows the first images.
 13. The methodaccording to claim 11, wherein the second images are enlarged in thesecond screen.
 14. The method according to claim 11, wherein performingthe second object recognition includes determining one or more candidatearticles, and the candidate articles are also displayed in the secondscreen.
 15. The method according to claim 14, wherein one of thecandidate articles is selectable for registration through the secondscreen.
 16. The method according to claim 14, further comprising:transmitting a result of the second object recognition to apoint-of-sale (POS) terminal.
 17. The method according to claim 14,further comprising: when no candidate article is determined, displayinga third screen indicating a button to retry the second objectrecognition.
 18. The method according to claim 11, further comprising:before performing the first object recognition, displaying a presetscreen showing a plurality of buttons for selecting an article forregistration.
 19. The method according to claim 18, wherein the firstscreen is displayed in place of the preset screen when the code symbolis read.
 20. The method according to claim 19, wherein the preset screenis displayed in place of the first screen after the button in the firstscreen is selected.